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Abstract 

A  control  system  is  being  provided  for  PBFA-I,  an 
operational  accelerator,  to  control  subsystem  functions 
such  as  sequencing  of  oil,  water,  gas,  and  vacuum  controls, 
to  coordinate  charging  and  firing  sequences,  and  to  control 
safe  and  idle  nodes  of  the  machine.  Important  design 
concepts  in  automating  this  system  are:  reliability, 
flexibility,  and  operability. 

Reliability  is  a  primary  criterion  for  executing 
single  pulsed  power  and  physics  experiments  as  well  as  for 
supporting  extended  experimental  series  without  a  failure 
that  would  cause  downtime.  Flexibility  to  respond  to 
changes  in  operating  parameters  and  experimenter  needs  is 
being  implemented  in  two  major  areas:  a  modular  I/O  system 
to  provide  hardware  flexibility,  and  structured  software  to 
ease  software  maintenance.  Operability  is  increased  with 
user-friendly  software,  and  automatic  sequencing. 


introduction 

The  Particle  Beam  Fusion  Accelerator  I  (P8FA  I)  is  a 
30-terawatt  machine  with  36  modules  that  has  been  in 
operation  for  almost  5  years.  It  is  housed  in  a  facility 
with  two  small  accelerators  that  share  data  acquisition  and 
some  process  control  systems.  The  control /monitor  (C/M) 
system  is  responsible  for  the  facility  oil  and  water 
systems  (the  oil  system  has  over  200,000  gallons  of 
transformer  oil  and  the  water  system  has  over  160,000 
gallons  of  deionized  water).  C/M  is  also  responsible  for 
the  two  PBFA  I  insulating  gas  systems  (SF6  and  synthetic 
air),  hydraulic  transfer  switches,  charging  system,  and 
timing  and  firing  systems. 

In  the  5  years  of  operation  as  a  driver  for  fusion 
experiments,  the  criteria  for  an  acceptable  control  system 
has  changed  considerably.  As  an  example,  PBFA  I  was 
brought  on-line  in  a  completely  manual  mode.  Semi¬ 
automatic  and  automatic  control  modes  were  added  later.  In 
comparison ,  PBFA  II,  a  100-terawatt  successor  to  PBFA  I 
currently  under  construction  cannot  be  operated  without 
computer  control.  At  the  time  of  its  construction,  PBFA  I 
was  the  largest  accelerator  Sandia  had  built  or  operated. 
What  we  have  learned  about  operating  and  controlling  such  a 
large'  accelerator  has  been  incorporated  Into  the  design  of 
the  PBFA  II  control  system. 

Baseline  Status 

The  original  proposal  for  automating  the  PBFA  I  C/M 
system  called  for  automatic  staging  of  the  accelerator 
conmencing  10-12  hours  before  the  shot.  However,  the 
original  system  was  too  dependent  on  knowing  how  long  each 
step  in  the  preparation  sequence  would  take.  The 
preparation  sequence  timing  for  this  RAD  facility  was  never 
predictable  and  the  sequence  changed  often.  So  a  more 
flexible  system  was  required;  one  that  was  not  dependent  on 
a  set  sequence  of  events  and,  yet,  easy  to  operate. 

Hardware  And  Software  Baseline 

Two  years  ago  the  C/M  system  was  a  manual  control 
system  (push-buttons  and  relays)  that  worked;  it  required  a 
minimum  of  two  operators,  preferably  three,  and  a  shot- 
coordinator  sequence  operations.  Custom-built,  8080-based 
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subsystem  controllers  were  available  and  the  programs  for 
them  were  in  various  stages  of  completion.  All  controllers 
were  communicating  with  the  host  computer  (an  HP-1000)  and 
monitoring  the  various  sensors.  But  only  one,  the  SF6  gas 
controller,  was  partially  running  in  semiautomatic  mode, 
and  that  operation  soon  ended  when  the  parameters  were 
changed.  There  were  many  programs  in  the  host  computer  to 
support  the  automatic  control,  but  they  were  30  closely 
tied  to  the  system  configuration  and  operating  parameters 
that  only  the  monitoring  programs  were  usable.  Figure  1 
shows  the  hardware  configuration,  the  dashed  lines  separate 
where  control  of  the  system  resided  in  the  manual, 
semiautomatic  and  automatic  modes. 


Figure  1 .  Hardware  Configuration 

The  computer  and  the  8080  controllers  supported  the 
manual  mode  with  monitor ing,  but  if  a  controller  or  the 
computer  were  not  available  for  the  diot,  manual  control 
was  used.  The  manual  mode  capability  has  remained  in  place 
as  a  backup  vh  lie  the  automatic  and  semiautomatic 
capabilities  were  added.  Figure  2  shows  a  simplified 
software  configuration.  The  Monitor,  Charglrg  Graphics, 
Operator  Interface,  and  Record  Programs  are  real-time  and 
active  during  the  charging  and  firing  sequence.  The  other 
programs  and  Update  and  Report  Generator  are  run  whenever 
convenient. 
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CAMAC  modularized  unit  was  written  and  reviewed.  Another 
proposal  was  to  replace  the  subsystem  controllers  with  a 
Hewlett-Packard  programmable  controller.  At  this  time, 
neither  proposal  has  been  initiated  due  to  funding 
availability.  The  CAMAC  system  offers  a  major  advantage  in 
that,  being  modular ,  it  may  be  bought  and  implemented 
piecemeal  as  funds  become  available  or  the  needs  become 
greater . 

To  bring  flexibility  into  the  software  and  make  the 
functions  performed  by  the  8080  controllers  less  hardware 
dependent,  decision-making  functions  were  moved  up  into  the 
host  computer  for  automatic  mode  operation.  The  8080 
controllers  are  reduced  to  I/O  processors  for  the  host  and 
perform  simple  I/O  operations  or  short  preprogrammed 
sequences. 

The  controllers  3till  retain  the  capability  to  perform 
ln-semlautomatio  mode  without  the  host  computer.  If  the 
8080  controllers  are  replaced,  the  1/0  processing  function 
would  be  the  most  hardware  Independent  and  have  the  least 
impact  on  the  automatic  system  software. 

Design  &  Implementation 

After  establishing  criteria  and  a  preliminary  design 
for  an  automatic  control  system,  we  proceeded  with  both  the 
human  factors  redesign  of  the  control  room  and  the 
programming  of  the  the  8080  controllers  for  the  semi¬ 
automatic  mode.  We  started  with  the  fluid  transfer 
systems,  bringing  these  into  semiautomatic  mode  after 
considerable  testing.  The  time  required  to  complete  this 
activity  can  be  broken  down  into  three  parts:  learning  the 
existing  software,  writing  new  or  revising  old  software, 
and  testing-debugging.  The  learning  took  approximately  30% 
of  the  cycle;  writing,  20% ;  and  testing  50?.  Once  on-line, 
these  controllers  freed  one  operator  on  the  C/M  team  from 
the  time  consuming  job  of  monitoring  the  fluid  transfer 
control  panels. 

Problems  &  Solutions 
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Figure  3.  Control /Mon itor  Room  Human  Factors  Redesign. 


In  the  process  of  bringing  the  fluid  transfer 
controllers  up  to  semiautomatic  node,  we  found  problems  in 
documentation  that  resulted  from  changes  made  in  the  system 
without  updating  the  drawings  and  schematics.  We  have  now 
established  a  set  of  as-built  drawings  that  are  up-to-date 
and  have  supplemented  them  with  notes  and  comments. 

We  also  found  design  errors  in  the  original  hardware 
in  sections  that  had  not  been  tested  until  the  controllers 
were  in  semiautomatic  mode.  We  found  during  this  program 
testing  phase  that  a  "debugger,"  whether  a  program  or 
hardware,  was  an  invaluable  tool.  With  It,  we  could  trace 
1/0  signals  from  the  bus  decoder  out  to  the  field  wiring. 
Also,  we  could  single-step  through  a  program  checking 
memory  locations  as  often  as  we  wished.  The  debugger  we 
used  saved  many  hours  of  testing,  changing  code, 
recompiling,  and  loading. 

Human  Factors  Implementation 

Bach  of  the  other  subsystem  controllers  were 
sequentially  programmed  and  tested.  When  controllers  were 
running  in  semiautomatic,  the  human  factors  redesign  was 
completed  with  the  installation  of  a  new  console  designed 
to  be  used  by  a  single  operator.  The  human  factors 
designed  console  put  radio  and  P.A.  system  access  at  the 
operator’s  fingertips  for  easy  communications.  The 
critical  manual  controls  were  paralleled  to  the  console. 
Information  readouts  on  the  8083  controllers  were  moved  in 
the  racks  so  they  could  be  viewed  from  the  console. 


Host  System  Modification 

The  monitoring  programs  running  on  the  host  computer 
were  wasteful  of  resources  and  required  considerable 
operator  attention.  These  were  modified  to  query  the 
system  memory  for  the  needed  parameters  Instead  of  asking 
the  operator.  The  disc  file  space  usage  was  reduced  to  1/1 
of  the  former  usage  by  dropping  the  transfer  and  storage  of 
large  blocks  of  empty  memory.  Then  CPU  usage  was  reduced 
so  that  other  control  programs  could  run  concurrently.  The 
host  system  was  upgraded  to  use  FORTRAN  '77  (a  language 
better  suited  to  interactive  use  that  FORTRAN  ’66),  and  the 
system  memory  wa3  expanded  to  allow  the  nulti-program  use 
that  would  be  required  with  the  automated  system.  Figure  2 
is  a  simplified  software  block  diagram  at  this  phase.  The 
real-time  programs  are  active  during  the  shot  charging  and 
firing,  vhile  the  others  operate  after  the  shot,  whenever 
convenient.  During  this  period  we  also  gained  experience 
operating  the  machine  in  a  semiautomatic  mode. 

By  this  time,  we  had  found  that  some  of  the  procedures 
programmed  into  the  semiautomatic  node  were  overly 
conservative,  causing  shots  to  be  aborted  vhen  the  problem 
was  not  severe,  and  some  programs  had  subtle  "bugs"  not 
severe  enough  to  cause  damage  but  noticeable  and  sometimes 
annoying. 

Software  Design 

The  design  for  the  "Triggering  and  Firing  (T&F) 
Program"  was  finalized  using  this  experience.  The  8080 
controllers  needed  to  be  reprogrammed  (their  programs  were 
in  EPROM)  to  be  only  1/0  processors  When  in  the  automatic 
mode;  their  commands  to  begin  short  set  sequences  or  single 
actions  would  be  sent  from  a  "CONTROL"  program  ( see  Figure 
4).  The  Monitor  program  will  continue  to  do  the  sane  thing 
that  it  had  always  done;  instead  of  controlling  its  ovm 
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Figure  2.  Original  Software  Configuration 


Operability  Baseline 

The  operation  of  the  system  in  manual  mode  required 
considerable  time  and  training  and  was  still  prone  to 
errors,  due  to  the  large  number  of  tasks  the  operator  must 
perform.  Not  only  must  there  be  an  operator  in  the  control 
room,  but  fluid  transfer  controls  had  to  be  monitored  in 
another  part  of  the  building.  The  two  minutes  immediately 
before  the  shot  while  the  capacitor  banks  were  charging 
were  particularly  stressful  to  the  operators,  and  many 
mistakes  occurred  during  this  time.  While  operating  In 
manual  mode,  changes  in  operating  procedures  from  different 
experimenter  needs,  and  changes  to  the  accelerator  hardware 
made  the  task  of  providing  design  criteria  for  a  control 
system  Increasingly  difficult. 

Design  Criteria 

A  C/M  design  team,  comprised  of  the  C/M  coordinator, 
the  shot  coordinators,  the  facility  supervisors,  and  the 
technical  supervisor  started  the  design  of  an  automated 
system  by  defining  the  general  criteria  and  answering  the 
following  questions:  what  do  we  want  the  operations  to  be 
like;  how  many  operators  should  it  take;  how  reliable  does 
it  have  to  be;  and  how  flexible  does  it  have  to  be? 

Operability  Criteria 

We  attacked  the  question  of  operability  first.  The 
number  of  operators  we  had  was  four;  two  on  each  shift, 
with  an  overlap,  so  all  were  often  available  at  Slot  time. 
We  wanted  to  reduce  the  number  of  operators  required  to  a 
minimum  and  also  wanted  the  control  room  operation  to  be 
simple  enough  to  train  new  operators  in  a  matter  of  8-10 
weeks,  instead  of  6  months  to  a  year  with  the  original 
system.  We  decided  to  design  a  system  vhioh  wculd  allow  a 
single  operator  to  fire  a  shot.  That  decision  meant  two 


actions  had  to  be  taken:  1)  the  control  system  had  to  be 
automatic  or  semiautomatic;  and  2)  the  control  room  had  to 
be  redesigned  based  on  human  factors.  The  automation 
design  and  the  human  factors  design  affected  each  other 
since  the  programs  that  needed  an  operator  response  had  to 
be  incorporated  into  the  control  room  design.  Thus,  the 
oontnol  room  design  assumed  the  semiautomatic  operating 
mode  to  be  the  normal  operating  mode. 

Reliability  Criteria 

Reliability  was  an  Important  consideration  but  not  in 
the  traditional  sense.  From  the  nuclear  power  industry, 
the  most  reliable  system  is  one  that  operates  continuously 
without  failure.  The  computer  would  have  an  uninterrupt- 
able  power  supply,  and  controls  would  probably  have 
redundant  backups.  Our  application  didn't  need  that  level 
of  reliability;  we  could  tolerate  short-term  failures  on  an 
infrequent  basis  but  not  long-term  failures  that  would 
prevent  executing  a  shot.  Therefore,  the  subsystem's 
controller  programs  were  designed  to  protect  the  system,  to 
shut  down  in  situations  they  did  not  recognize,  and  to 
return  control  to  a  human  operator,  thus  avoiding  damage  to 
major  pieces  of  equipment.  Tested  spares  for  many 
components  are  kept  on  hand  to  facilitate  rapid  recovery 
from  failures.  The  host  computer  was  put  under  a  4-hour 
response  service  contract,  and  we  kept  spare  PC  boards 
critical  to  our  application.  The  weak  hardware  lilnk  is 
the  subsystem  controllers.  Bach  has  a  processor  board,  an 
I/O  board,  and  a  memory  (PROM)  board  that  are  identical, 
but  each  Is  completely  customized  for  its  application.  The 
I/O  board  is  also  a  custom  board  that  is  not  supported  by 
the  original  supplier,  and  there  are  no  spares.  This 
situation  led  to  a  proposal  (discussed  later)  to  replace 
the  8080  subsystem  controllers  with  other  hardware.  We 
kept  the  reliability  of  the  system  high  by  not  requiring 
the  computer  or  all  the  subsystem  controllers  to  be  on-line 
for  a  shot.  By  retaining  the  manual  control  capability,  an 
experienced  operator  could  work  around  a  "down"  controller, 
though  he  would  require  assistance. 

Flexibility  Criteria 

Flexibility  was  another  main  criterion.  PBFA  1  not 
only  is  a  fusion  experimental  driver  but  is  also  an 
experiment  in  Itself.  The  driver  experiment  has  been  to 
refine  existing  operating  techniques,  try  new  ones,  and 
find  the  best  ways  to  operate  a  large  accelerator.  In  this 
continuing  experiment,  operating  parameters  change.  For 
example,  the  nine  Marx  pulser  units  (MPUs)  that  trigger  the 
36  Marx  generators  were  individually,  tested  and 
characterized  by  their  manufacturer.  The  recommended 
operating  parameters  were  80  KV  charge  and  80  psig  gas 
switch  pressure.  We  found  that  by  increasing  the  charge 
and  reducing  the  pressure,  the  MPUs  as  a  system  had  50% 
less  spread.  The  original  control  software  had  not  allowed 
for  these  kinds  of  changes.  To  implement  a  new  control 
system  with  the  same  difficulty  in  incorporating  changes 
would  be  foolish.  The  new  system  therefore  has  flexibility 
designed  in  hardware  and  software. 

Hardware  Flexibility 

In  the  hardware,  vhere  we  must  use  the  8080-based 
controllers,  we  have  added  a  CAMAC  [1]  Crate  to  the  timing¬ 
triggering  system  to  allow  us  to  program  delays  and  trigger 
the  accelerator  from  the  host  computer.  While  we  were 
bringing  the  timing  system  interface  on-line,  we  found  a 
multitude  of  modules  available  for  CAMAC.  A  brief 
literature  survey  showed  us  that  we  could  add  process 
control  functions  to  the  host  computer  through  these 
modules.  The  first  addition  was  the  control  of  an 
experimenter's  capacitor  bank.  An  analogue  input  module,  a 
digital  input  module,  and  a  digital  output  module  gave  us 
sufficient  capability  to  control  the  capacitor  bank 
charging,  voltage  level,  and  safing.  So,  a  proposal  to 
gradually  replace  some  or  all  of  the  8080  controllers  with 
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sample  rate,  the  nonitor  and  control  programs  now  will  be 
coordinated  by  the  T&F  program.  The  Report  Generators  and 
UPDATE  program  are  not  shown  in  Figure  4  but  continue  to  be 
used.  The  T&F  program  will  handle  the  operator  interface, 
coordinate,  schedule  the  corollary  programs,  and  do  error 
handling.  The  T&F  program  would  have  two  modes,  an 
interactive  ncde  that  was  used  mainly  for  testing,  and  a 
command  file  mode  that  consisted  of  sets  of  commands 
already  translated  by  a  Command  Validator  Program  into  the 
form  that  could  be  sent  by  the  control  program.  These 
command  files  would  be  created  by  the  operator  before  the 
shot  and  would  have  various  functions,  matching  the  various 
types  of  shots  and  shot  preparation. 


Summary 


Moving  from  plan  and  design  to  implementation  is 
always  the  most  difficult  part  of  a  project.  We  have  moved 
forward  with  the  designed  upgrades  intermittently.  The 
semiautomatic  mode  worked  very  well,  with  the  exception 
that  it  was  not  flexible  enough.  A  parameter  or  procedure 
change  meant  testing  a  new  program  and  burning  new  PRCMS. 
This  made  some  changes  difficult  since  testing  some 
programs  required  firing  the  accelerator.  Program 
priorities  reduced  our  resources  and  has  delayed 
implementation. 

The  PBFA  I  Control  system  is  presently  being  operated 
in  a  semiautomatic  mode,  with  work  slowly  continuing  toward 
the  completion  of  the  designed  automatic  system. 
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Figure  4.  Automatic  Software  Configuration 


Operation  in  the  automatic  mode  would  not  exclude 
operator  intervention.  For  instance,  if  during  the 
execution  of  the  "Full  Shot"  command  file  a  situation 
occurred  that  required  operator  attention,  an  error  message 
would  be  sent  to  the  Error  and  Status  terminal,  the 
operator  terminal  would  get  a  menu  of  possible  actions,  and 
the  sequence  would  go  into  a  HOLD,  waiting  on  an  operator 
command.  The  operator  can  then  Continue  from  that  point  in 
the  command  file,  Restart  the  comnand  file,  or  ABORT;  the 
last  would  start  another  command  file  to  take  the  machine 
to  a  safe  condition.  The  operator  could  also  initiate  a 
told  by  pressing  a  preprogrammed  key. 

This  design,  though  its  purpose  is  to  put  the  period 
from  10  minutes  before  to  5  minutes  after  the  shot  in 
automatic  mode,  is  flexible  enough  to  be  expanded  to  a  24- 
hour  facility  monitor. 


